How to Implement A Priori Information : A Statistical Mechanics Approach ∗

Generalization abilities of empirical learning systems are essentially based on a priori information. The paper emphasizes the need of empirical measurement of a priori information by a posteriori control. A priori information is treated analogously to an infinite number of training data and expressed explicitly in terms of the function values of interest. This contrasts an implicit implementation of a priori information, e.g., by choosing a restrictive function parameterization. Different possibilities to implement a priori information are presented. Technically, the proposed methods are non–convex (non–Gaussian) extensions of classical quadratic and thus convex regularization approaches (or Gaussian processes, respectively). Specific topics discussed include approximate symmetries, approximate structural assumptions, transfer of knowledge and combination of learning systems. Appendix A compares concepts of statistics and statistical mechanics. Appendix B relates the paper to the framework of Bayesian decision theory.